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ABSTBACT 

Tbis work bas been done as part of tbe effort to plan 
tbe Bational Institute of Education <BIE) • Tbe report, one of a 
series, describes a aanageaent plan recoaaended for tbe Basic 
Besearcb Division of tbe proposed Bational Institute of Edocaticn. 
Tbe plan was developed following a survey of aanageaent practices in 
a nuaber of federal agencies supporting basic researcb activity, and 
incorporates aany of tbe lessons learned tbrougb experience vitb a 
variety of aanageaent policies and practices. Tbe report presents an 
overview of recoaaended aanageaent activity, tbe underlying cbcices 
tbat were aade in tbe developaent of tbe aanageaent plan, and tbe 
alternative cboices tbat would bave resulted in alternative plans. 
Also provided is an evaluation of tbe recoaaended aanageaent plan 
beoed oh criteria relevant to tbe quality of tbe supported BSC and 
tbe effectiveness oi* tbe aanageaent procedures related to required 
aanageaent skills, cost, tiae, and tbe ability to attract a 
supportive constituency. <Autbor/DB) 
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PREFACE 

This working note contains a description of a managenent plan recon^""*" 
oended for the Baaic Research Division of the proposed national Institute 
of Education. The plan was developed following a survey of t&anagement 
practices in a nuober of federal agencies supporting basic research 
activity* and incorporates iDany of the lessons learned through the 
experience with a variety of taanageoent policies and practices. 

The plan was developed in support of the work of the National 
Institute of Education Planning Unit » responsible for the preparation 
of an InterltB organization and aanageaent plan for the entire institute. 
Consequent ly » the management plan described here is consistent with the 
Interim Organization document released by the Planning Unit in April » 
1972. 

Material for the preparation of the Basic Research Divittlon's 
management plan was taken primarily from one of a series of Rand 
documents concerning the proposed National Institute of Technology: 
UN*7676 entitled '^National Institute of Education: Methods for Managing 
Fundamental Research." Others In the series include: 

o National Institute of Education: Preliminary Plan for the 
Proposed Institute (R-657-HEW), 

o National Institute of Education: Methods for Managing Practice- 
Oriented Research and Development (WN- 76 7 7- HEW) , 

o National Institute of Education: Methods for Managing Program* 
matic Research and Development (WN-7678-HEW) , 

o National Institute of Education: Organizational and Managerial ^ 
Alternatives (WN-7679-HEW) , 

o National Institute of Education: Evaluation of Methods for 
Managing Research and Development (WN-7680-HEW) . 

This series of reports seeks to provide a basis for research Into 
Improved management practices for non-mllltary federal R&D, The principal 



-Iv- 



purpose of iheiio reportq, however la to u. . 

K K Liie ait s RiD oanaReoent procedures. 
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I . IN TK) DUCT ION 



PURPOSE OF THIS REPORT 

This report describea a aatiagcnent plan for the Basic Research Diviflion 
of Che Nacional Institute of Education as described in the Interim Organiza- 
tion document of the NIC Planning Unit, April IS, 1972. 

The plan is limited to the management of what has come to be known 
as fundamental or basic research. This is research undertaken primarily 
to add to knowledge about social, group, and individual behavior funda* 
mental to education, rather than to further program goals or improve 
Immediate practice. Basic research is conducted according to disciplined 
{Standards of inquiry that determine what experimental data are acceptable 
and by what means conclusions may be derived from stated assumptions 
or demonstrated facts. The standards of inquiry are established and 
preserved by che scientific community itself and each researcher's pro- 
fessional status is determined by the quality of his work as Judged by 
his scientific peers. 

Since progress in basic research is guided more by consideration of 
where solutions are possible and where intuition hints that new results 
are likely to be found than by the immediate practical value of results, 
basic research directions are ordinarily chosen primarily by the researchers 
themselves. This does not mean that practical problems can have no influ- 
ence on the directions taken in basic research. Entirely new lines of 
research can be started in response to a practical need, and levels of 
effort allocated to different areas of inquiry can be adjusted to reflect 
their potential relevance to practice. 

TYPES OF MANAGE>ff:NT ACTIVIT/ 

To facilitate presentation, the recommended management plan will be 
broken into three types of activity: 

o Program planning, 

o Program development, 

o Program evaluation. 



These categories are deliberately chosen to group together qualitatively 
similar toanagcnent activities. 

Program PloK^^ing includes all the actions taken to foster, detect, 
and Incubate research in new and ill-fortned areas of Inquiry. Also included 
are the procedures for deciding viiich new researcli areas will be added 
to the set of ongoing streams of activity, and the priorities for budget 
allocation among all areas of research considered for funding. 

Pvogram Development is managing the continuous process of refining 
and elaborating knowledge in an area. As a management process. Program 
Development is typically, though not always, an iterative and continuing 
sequence of stages involving: 

o Assessment of research needs, 

o Generation project ideas, 

o Selection of projects to support, 

o Monitoring of project performance, 

o Evaluation of project outcomes, and 

o Utilization of results. 

In various management paradigms, these stages are managed in different 
ways. Sometimes they are performed by program directors, sometimes by panels 
of scientists, and sometimes not at all. In most R&D, and especially 
when basic research is be ng done, action proceeds concurrently in several 
of the stages. This activity is called ProRram Development because from 
program management *s perspective, basic research is continuously developing; 
it is evolutionary and expositional in nature: the goal of research is 
adding fine structure and precision to a basic idea, and this happens 
through a sequence of project generation, project selection, and outcome 
events chat develop over a long period of time. 

Frcgvan Evaluation is the assessment of what has been accomplished 
at some point in time by the researchers in a knowledge area, and the 
judgment of what ought to be doner next. 

Discussion of recommended management methods will focus on what is 
done at the progran: director *g level, and only occasionally at higher 
levels . 



PLAN OF THE REPORT 

The presentation of the recoianended management plan for the Division 
of Basic Research will be divided into three sections. Section II presents 
an overview of recommended management activity, in successively deepening 
detail. Section HI presents the underlying choices that were made in 
the development of this management plan, and the alternative choices 
that would have resulted in alternative management plans. Finally, 
Section IV provides an evaluation of the recommended management plan 
according to criteria relevant to the quality of R&D supported and the 
effectiveness of the management procedures related to required manage- 
ment skills, cost, time, and the ability to attract a supportive constituency. 
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II. OVERVIEW OF THE recommend::o management plan 



Four major alternative management procesBes have been identified 
and evaluated as candidates for the Division of Basic Research. These 
have been named: Dual Review, Single Panel Review, Mail Review, and 
No Review. The plan recommended in this report has charar tcr^ '^tics 
^.iken predominantly from the Mail Review paradigm, though cnara* loris- 
ti::s from the other paradigms have also bee^ utilized, basic advan- 

tages and shortcomings of the Mail Review paradigm whict. recommended 
for the Basic Research Division of the National Institute of Education arc 
described in Section III of this report. 

Mail Reviews are currently used in many places ii tu^i federal 
government, but particularly in the Engineering, MiLh^tu.. il cal and 
Physical Sciences divisions of the National Science Foundation. Exper- 
ience with Mail Reviews extends back to the Foundation's Inception and has 
been considered Lo be a satisfactory procedure by both Foundation manage- 
ment and by the scientific communities affected. 

This plan does not recommend a complete duplication of NSF's Mail 
Review procedures. Instead, it describes the basic Mail Review paradigm 
modified (1) to be more responsive to the kinds of problems found in 
education, and (2) to maintain compatibility with current procedures used 
in funding the educational research projects that will be transferred from 
the Office of Education to the National Institute of Education. The 
essential features of the recommended management plan are: 

o Support is to be given to individuals or institutions for work 
on individual projects that have been proposed. 



J. \fitt:, A. Lieberman, and R. Levien, National Institute of Education: 
Methods for Managing Fundamental Research , WN-7676-kEW, The Rand Corporation, 
November 1971 . 

The Mail Review procedures were invented and used by Wayne Gruner, who is 
now Senior Staff Associate, Research Directorate, National Science Foundation, 
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o Support is aDOPded for a limited period of tvne^ but can he 

renewed through presentation and approval of a new application. 

o Proposals are evatuated by obtaining mailed oormente from a 
number of scientists expert In the area of the proposal. 

o The majority of managerial effort is devoted to eeleating a 

eet of projeote thought to balance moet effectively the various 

lines of research being pursued, 
o Evaluation of reeearch aotixnty Is conducted periodically by 
convening panels of outside scientists who review the list of 
active projects. 



GENERAL CHARACTERISTICS OF SUPPORT 
Mechanism of support: 



Managerial emphasis : 



Finite duration project grants would be made 
to principal Investigators of Independent 
research projects. Generally, these 
scientists will be employees of a university 
or non-profit firm. A small amount of 
foraula support may also be awarded 
to the grantee's institution. 

Moat managerial effort should be devoted 
to selecting the set of projects thought 
by division management to balance 
most effectively the lines of research 
being pursued by the institute. 



Program Planning 

Role of Planning: Planning in the Basic Research Division 

should be concerned with setting allocation 
priorities among the various research 
areas to be funded by the Division. 
Within priority areas » clusters of projects 
may be grouped under a program director. 
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Type of Planning: Primary planning activities would include 

travel > conferences , and workshops to 
promote communication among the research 

communities concerning the selection 

of priority areas among fields of research. 



Program Development 
Sources of project Ideas: 



Means of proposal review: 



Procedures for the 
selection of projects: 



Scientists primarily cut *^ ^ae the insti- 
tute would be funde'* to work on problems 
that they hav<* -efined. Occasionally ^ 
a scienti*. s interest in a priority 
arc? '-xght be stimulated by contact with a 

rograo director. These program 
directors would offer advice and 
technical assistance to project appli- 
cants , 

Each proposal would be sent to a group 
of specialists in the field of inquiry 
for mailed comments on the technical 
quality of the project idea and plan, 
the reputation of the principal investi- 
gator » and the availability of needed 
research facilities. Returned comments 
would then be compiled and sent back 
to each reviewer for additional comments 
and rebuttals. 

The program directors would discuss 
priorities and research opportunifes 
with scientists and educational practi- 
tioners throughout the year. They would 
also interact with special panels of 
outside scientists who review their 
piograms annually or semi-annually. 
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At the budget allocation time, occurring 
at regular intervals of one yeau or six 
months, the program director would 
choose the set of projects that his 
discussions and previous mailed reviews 
have convinced him offer the best 
combination of research opportunities. 
This method would allow him to compare 
the merits of competing proposals 
against each other, rather than to select 
prcpcs<als for funding independently of 
each other. 

Monitoring of project Substantive progress on projects would 

performance: noL bo closely monitored; though progress 

■ reports and site visits for major projects 
would be required. Monitoring activity 
would be conducted by project officers 
withii-i the division and would be ^-.rimarily 
intended to provide tecltnical assistance and 
communication to the R&D performers on 
events and advances within other projects , 
.•and feedback information to the program 
directors on promising research areas or 
needs identified during the course of 
project activities. 

Evaluation of project Projects would not be subjected to for- 

outcomes: mal evaluations. Instead, the Project 

Officer, with consultation from out- 
side scientists where needed, would 
provide some assessment on the quality 
and potential utilization of research results 

to program directors concerned with 
future allocation priorities and 
stimulation of proposals in areas of 
promise • 
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Progtam Evaluation 
Means of cnraluation: 



Timing of Che evaluation: 



Purpose or laplementatlon 
of evaluation results: 



A panel of outside scientists , permanently 
const 1 tut ed» would meet to review the 
selection decisions made by each pro- 
gram director after awards have been 
granted. The format would be a round- table 
discussion during which panelists would ask 
qucctlono regarding quality of research 
supported, relevance of projects to pro- 
grams of the Institute, and balance among 
fields supported. Each panel would 
review all the projects within a 
specified proorac area. 

A review session would be held semi- 
annually! shortly after each period 
during which project grants are awarded. 

Program directors would utilize the 
results of the panel reviews as a 
guide In determining future allocation 
priorities . In addition, the reviews 
would help communicate to the research 
and practitioner communities the results 
and contributions of UdD supported by 
the institute. 



DETAILED DESCRIPTIONS OF MANAUEMENT ACTIVITIES 

Within this management plan, prograii directors do not Interact with 
outside scientists, convened In panels, to decide jointly on project 
selections. Instead, the directors use repreaentatives of the research 
community as consultants, providing assessments on the individual merits 
of project proposals regarding technical quality, reputation of the 
principal investigator, and availability of research facilities. 
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Comparlsons among candidate projects competing for funds are made by the 
program directors themselveSt and are judged subsequently by panels of 
outside scientists convened periodically to review the entire mix of 
active projects. In this vay» the staff of the Basic Research Division 
can provide direction and promote concentration of effort to selected 
research areas of high priority. 

Program Planning 

Programs of the Basic Research Division would actually be clusters 
of projects related by discipline, and not ^collections of logically 
Interrelated projects deduced from a set of overall goals* Thus, ^^fields 
of inquiry" might be a better name than "programs" for these units of 
activity. Especially important new ideas that seem likely to develop 
into a distinctive field of Inquiry would be nurtured by the program staff 
through the usual means: traveling , workshops , conferences , and general 
- ^ersuas ion # Some respo.isiblllty for detecting and encouraging these 
new ideas should lie with the program directors; but in addition, a senior 
program director reporting at the top management level may spend all his 
time searching out and stimulating nascent fields of inquiry which offer 
significant potential. His principal means of project stimulation would 
be 'he same as the program director's. 

Program Development 

At intervals throughout the year, the program directors would visit 
scientists in thoir fields of inquiry, discussing new trends, important 
events, and research options. These discussions would help the program 
directors form their scientific priorities, and keep them informed; but 
also serve to stimulate research proposals. 

Proposals would be received in the program director's office through- 
out the year from individual investigators. The program director would 
sort through the stack of received proposals and select those that seem 
to qualify for review in terms of preparation of study plan, budget, and 
personnel specifications. Each proposal selected for review would be sent 
to four or five specialists for a written review of its merits. The 
specialists would be chosen to match the subject area of the proposal and 
would be given guidelines on the important criteria to consider in writing 
Q their review. 

ERIC 
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The program director would select these reviewers from a list kept as up 
to date as possible. If the proposal is particularly complicated or if it 
receives tnl:<ed reviews initially, additional reviews may be solicited 
from other members of the list. The first round of replies would be 

summarized and mailed to each first round reviewer, who might then revise 
his original review and send it ba:-k to the program director. Upon 
receiving all the second round reviews, the program director would have 
an expert, external assessment of a proposal's worth. 

Two or three times a year,, the program director would examine the 
backlog of reviewed proposals and select a subset of them which fits 
bis budget and which he believes to be an "optimal" balance of innova- 
tive ideas, normal science progress, special opportunities, and chances 
for development of new research talent in the lines of research he is 
pursuing. The budget of some proposals may be altered to eliminate weak 
portions, or increase strong ones. In these cases, the program director 
would negotiate with the applicants concerning these changes. The program 
director's final choices would then be reviewed and approved by his 
organizational superiors . 

Grants would be awarded for a fixed amount of money; expendable over 
a specified period of time, usually less than a five-year period. During 
this time, annual progress reports would be requi , but unless major 
difficulties arise, the investigator would be free to choose his course. 
An investigator who is not making reasonable progress may be given one 
year's notice, prior to a final decision to revoke his grant, but this 
should occur only infrequently. Project outcomes would not be evaluated 
on a formal basis. 

A flow chart of this proposed process appears in Fig. 1. 

Program Evaluation 

Programs would be evaluated by a panel of scientists, specifically 
appointed for the purpose. The panel would convene shortly after each 
period when project awards are made to review the list of projects 
selected and rejected, and cross-examine the program directors on their 
decisions. The panel would evaluate all the programs managed by a section 
of the division, usually encompasing from three to six programs. 

The meeting format would not be highly structured, but rather informal. 
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The chairman o£ the panel, elected hy the panel menibers tKexoselves. would 
ask each program director In turn to describe briefly his project selection 
decisions. Following this presentation, the meeting would be opened for 
round-table discussion. The attitude, of the program staff shguld be to '/throw 
open the files, and let the panel see any thing they want." Back and 
forth, discussion on each program would proceed for a while until all 
opinions are on the table. The panel could then vote a resolution if 
desired, but usually the spoken comments of the members would constitute 
adequate evaluations. 

These evaluations would be reflected in future decisions only if the 
program directors wished to follow them. Scieiitlsts' opinions about the 
abilities of a program director would easily find their way back to the 
program director's supervisors. In addition, the section supervisor 
would attend the panel meeting, and talk with the panelists on an 
individual basis whenever the need arises. 

Staffing 

Progrgm Directors. With rare exception, all would have Ph.D.s or the 
equivalent. The turnover rate should be limited to one departure per 
approximately two positions, for half of the staff, and one departure per 
approximately eight years for the rest. Each program director would be 
selected by his immediate suprevisor. 

Mail Reviewers . Mail reviewers would be selected by the program 
director with the objective of obtaining a match between topics in the 
proposed research and specialties covered by the reviewers. The list 
from which reviewers are drawn should be frequently updated, and 
sufficiently long that only rarely would any reviewer see more than three 
proposals per year. A great deal of effort must be expended to build and 
maintain the list of mail reviewers. Mail reviewers should be paid 
between $25 and $50 for each proposal reviewed. 

Panel. The program directors and their suprevisors would select 
replacement panelists from the coumunity of scholars, with approval 
required from the Division Director. The term of office should be for 
only two years, but there should also be no statutory lipiitation. The 
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panelists should span a greater range of xesearch interests than the 
programs evaluated, and may include applicativ ns-oriented researchers. 
Panelists should be paid for their services, at c. rate of approximately 
$100 per day. This would encourage them to read the proposals, and parti- 
cularly important, to familiarize themsel ves with the programs and 
priorities of the Division, 
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III. CHOICES MADE WHICR FORM THE BASIS OF THE MANAGEMENT PLAN 



ISSUES OF RESEARCH SUPPORT 

1. Alternative mechanisms of support representing different 
degrees of contractual specification: 

(1) Investigators are relatively free to pursue 
activity in the area of support without committment 
to producing an agreed product. Support of this 
kind is usually provided by means of a grant , though 
contracts may be used. 

(2) Researchers are committed to produce a defined product. 
Support of this kind is almost always provided by 
contract . 

Alternative chosen for this management plan: 
(1) Support predominantly through grants. 

2. Alternative means of dispensing basic research funds : 

(1) Support is provided on an independent project basis > 
either to a single principal investigator or to a 
predetermined project group. Actual funds may be given 
either directly to the project or to the university 

or institution of the investigator, in support of the 
particular project. 

(2) Support is given to institutions which support multi- 
•> project basic research activity, on a formula grant 

basis , or as core support for a number of individual projects* 
Alternative chosen for this management plan: 
(1) Support on an individual project basis. 

3. Alternatives concerning the duration of support: 

(1) Support is given on a continuing basis, contingent 
only on continuing productivity. 

(2) Support is given only for a limited period of time, 
but can be renewed through presentation and approval 
of a new application. 
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Alternative chosen for this management plan: 

(2) Support for a limited period of time, and renewable. 

PROGRAM PLANNING ISSUES 

1. Alternative amounts of planning activity engaged in by the 
Division of Basic Research: 

(1) No Planning. Unsclicited proposals are processed 
and selected for funding on the basis of their 
individual merits. 

(2) Planning activity is limited to the i dentification 
of ^ap areas of research. This information may then 
influence selection of unsolicited or stimulated 
proj ects . 

(3) Planning activity includes the determination of alloca- 
tion priorities . In this case, gap areas are identified 
as well as specific areas of research which are labeled 
as high priority areas. Project selection and stimula- 
tion may then be carried on in light of these allocation 
priorities . 

(4) Emphasis on planning. In this case, division planners 
identify th e specific research objectives and steps 
that are needed to achieve predetermined goals of 

the division. Grants or contracts are then let for 
the accomplishment of individual steps. 
Alternative chosen for this management plan: 
(3) Planning includes determining allocation priorities. 

2. Alternative forms of planning activity : 

(1) T ravel to universities and research institutions and 

to practitioner communities to establish communication 
on problems and needs for research. 

{?.) Cent ^re-ices of the p'^ ^actitioner community to discuss 

the role of research and development in solving practical 
problems, and to determine priorities for research. 

(3) Workshops of the research community to establish 
research priorities and communicate the results of 
R&D needs. 
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(4) Problem Analyses > Studies to determine the importance o 
educational problems, and to identify tbeir causes and 
potential solutions. 

(5) Program Analyses . Identification of the research steps 
needed to achieve program objectives. 

Alternatives chosen for this management plan: 

(1) and (3) as major planning activities for the Basic 
Research Division. 

3. Alternative performers of the planning activity (Who does 
the planning?) 
CI) Division Director . 

(2) Branch Cliiefs or Program Directors . 

(3) Project Officers , 

(4) Teams of internal staff of the Basic Research Division 
serving a planning role. 

(5) Teams of outside researchers called on an ad hoc 
basis. 

Alternatives chosen for this management plan: 

(1) , (2), and (3) for travel; 

(2) , (3), and 5^ for workshops among researchers. 

PROJECT IDEA GENERATION AND PROPOSAL DEVELOPMENT ISSUES 

1 . Alternative sources for project ideas : 

(1) Branch chiefs and Project Officers , 

(2) Project reviewers, 

(3) University or independent researchers . 

(4) Non-profit organizations or others contracted 
for project generation. 

Alternative chosen for this management plan: 

Greatest reliance on (3) with supplemental contributions 

from (1) and (2) . 

2. Alternative degrees of stimulation or encouragement for 
project generation in gap or high priority areas identified 
by planning activities: 
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(1) No stimulation . All project ideas are unsolicited and 
stem from outside the division. 

(2) Limited stimulation through advertisement of areas 
that the division considers high priority, ualng. 
announcements or notifications by the Division 

to the research community. 

(3) High degree of stimulation and encouragement through 
travel to research location, workshops of members of the 
research community, and advertisement of high priority 
areas of research in research journals . 

Alternative chosen for this management plan: 
C3) High degree of project stimulation. 
3. Alternative degrees of technical assistance given by the staff 
of the Basic Research Division to researchers in the preparation 
of project ideas and proposals: 

(1) No technical assistance , 

(2) Limited assistance through telephone calls on the 
strengths and weaknesses of proposals prior to their 
formal evaluation and/or following thair rejection. 

(3) Written reviews and technical assistance to authors of 
selected proposals prior to formal e-^'aluation, and /or 
following their rejection. 

(4) Written reviews and technical assistance to authors of 
all proposals prior to formal evaluation and/or following 
rejection. 

Alternative chosen for this management plan: 

(3) Reviews and assistance to selected authors of proposals. 

PROPOSAL EVALUATION AND SELECTION ISSUES 

1. Alternatives concerning the r elative ^amount of project selection power 
granted to representatives oi the research community ; 

(1) Total Power : In this case, project selection may be 

conducted by panels of peer scientists with no review 
or approval, power given to internal institute staff. 



(2) Major Power: In this case, representatives of the 
research counnunity actually determine the funding 
priority for proposals, and staff personnel act as 
consultants to the panels, screening proposals to 
be reviewed and adnlnlstering site visits. 

(3) Limited Power ; In this case, representatives of the 
research community serve as consultants to the Internal 
staff, either through mall reviews of project 
proposals, or through panel discussions to 'evaluate 
the technical quality of project proposals. Other 

e^valii^tions concerning relevance and project balance 
would. be conducted by the staf f v 

(4) No Power ; In this case, project selection is conducted 
entirely by the Internal staff, with no systematic 
procedures for evaluation by the research community. 

Alternative chosen for this management plan: 

(3) Limited project selection power given to the research 

community. 

Alternative criteria for project selection : 

(1) Technical qua lity of the project idea and plan^ 

(2) Reputation of the principal investigator , 

(3) Availability of research facilities for the 
conducting of the research plan. 

(4) Relevance of the project to ongoing programs and 
interests of the entire institute. 

(5) Training value of the project (bringing new people 
into areas where greater research is needed) . 

Alternative chosen for this management plan: 
No single crite)irlon; Instead, a balance among (2) through 
(5) with a required minimum rating in 
Alternative sourcea of proposal reviewers: 

(1) The research community outside the Institute. 

(2) The research staff within the institute. 

(3) The management staff within the institute. 

(4) Mixture of researchers from within and outside 
the institute. 
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(5) Mixture of researchers and management staff from 

within the institute. 
Alternative chosen for this management plan: 
(1) Research community outside the institute, 

4. Alternative procedures for the selection of projects ; 

(1) Internal staff approves proposals clearly desirable, 
rejects proposals clearly undesirable, and submits 
all remaining proposals to a panel of representatives 
of the research community who then review for technical 
qtialit> and make recommendations for funding or rejection. 

(2) Internal staff approves proposals clearly desirable, 
rejects proposals clearly undesirable, and requests 
written mail reviews for assessments of technical 
quality on all remaining proposals, then makes decisions 
on funding based on these mail reviews. 

(3) Requests for mail reviews on the technical quality 
of all proposals are made and serve as guides to 
funding decisions by the internal staff. 

(4) Panels are convened to review all proposals and 
make recommendations for funding to the internal 
staff based on the consideration of technical quality 
only. 

(5) Panels are convened to review all proposals and make 
recommendations for funding to the internal staff 
based on considerations of technical quality, relevance 
to ongoing programs and interests, and balance among 
projects approved for support. 

Alternative chosen for this management plan: 

(3) Mail reviews on technical quality for all pt(tpo^SL%^ 

qualified for review.. 

5. Alternative number of reviews given each proposals selected 
for review: 

(1) single review in which all asaessments are submitted 

independently to the internal staff for funding decislcma. 
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(2) Tvo-stage review in which evaluations of all reviewers are 
circulated among themselves so that revisions or rebuttals 
may be made prior to recommendations to the internal 
staff for funding decisions. 

(3) Dual review in which outside scientists review for 
technical quality and Internal management staff review 
for relevance and balance. 

(4) Dual review in which one review is held prior to funding 
decisions and one held subsequent to project completion 
for purposes of project evaluation. 

(5) Combinations of the above. 
Alternative chosen for this management plan: 

(2) Two rounds of mail review prior to project selection. 
6. Alternatives for the timing of proposal review and selection : 

(1) Group reviews are Initiated whenever a 
predetermined number of proposals have been 
received . 

(2) Group reviews are initiated at regular intervals 
such as annually or semi-annually. 

(3) Reviews are initiated individually whenever a 
proposal is received . 

Alternative chosen for this management plan: 
(2) Group reviews at regular intervals. 

PROJECT M3NIT0RING ISSUES 

1. Alternatives for what is monitored: 

(1) Fiscal audit. 

(2) Substance audit (progress and quality) . 

(3) personnel usage audit , 

(4) Compliance to contractual or agreed obligations. 

(5) Nothing is monitored. 

Alternative diosen for this management plan: 
Empuasis on (2) substance audit. 

2. Alternatives for who aonitors? 

(1) Internal management staff (Project Officers). 
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C2) Ad teaue of outside researchers . 

(3) Ad hoc teams of outside and Institute researchers , 

C4) Ad hoc teams of management consultants , 

(5) Subgroup of the reviewers who participated in the 

Project evaluation. 
Alternative chosen for this management plan: 
(1) Project Officers of the Internal management staff. 

3. Alternatives for the role of monitoring ; 

(1) To maintain compliance with contractual or agreed 
obligations . 

(2) To provide technical assistance where needed. 

(3) To provide communication on events and information 
to the research performers on relevant issues and 
advances . 

(4) To provide communication to planners and managers 
of the internal staff on events and advances in the 
field. 

(5) To collect information for subsequent project evaluation . 
Alternative chosen for this management plan: 

Emphasis on (2) > (3), and (4); assisting and communicating. 

4, Alternative degrees of monltorin|g activity : 

(1) Minimal monitoring; mostly by phone , 

(2) Required progress reports at regular intervals. 

(3) Site visits and progress reports at regular intervals. 

(4) Continual communication , written or verbal, on 
progress, problems, directions, and plans. 

Alternative chosen for this management plan: 
(2) For most basic research projects. 

PBOJECT EVALUATION ISSUES 

1. Alternatives on what is evaluated: 

(1) Nothing; final reports are merely entered into the 
project information system for subsequent retrieval 
by other researchers . 
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(2) Timeliness of output, 

(3) Quality of substi^nce of report. 

(4) Compliance t^ what was asked for or agreed up on. 

(5) Degree to which research Is Implemented within 
an operating program . 

(6) Degree to which research results are utilized 
by other researchers . 

(7) Effectiveness of management within the project. 
Alternative chosen for this management plan: 
Emphases on (3), and (6) stressing quality and 
utilization . 

Alternatives concerning who cunducts the. evaluations of projects 

(1) Project Officers or Internal staff of the tllvlsloa. 

(2) Project Leader or principal Investigator of the project . 

(3) Subgroup of the reviewers who participated In the 
evaluation prior to selection . 

(4) Ad hoc group of outside scientific reviewers . 

(5) Higher Institute management (Branch Chiefs, 
Division Director, or Advisory Council) . 

Alternative chosen for this management plan: 
(1) Project Officers with consultation from ad hoc 
groups of outside sciwitista where needed* 
Alternatives for the primary role of project evaluation : 

(1) Subsequent selection situations involving the 
same principal investigator. 

(2) Feedback for planning; information on progress or 
new problems arising in the pursuit of research 
objectives; i.e., future allocation decisions ^ 

(3) Cammunlcation to the research community on the 
identification and selection of new ideas and 
research needs; e.g., for project stimulation 
purposes. 

Alternative chosen for this management plan: 
All, but with an emphasis on (2) and (3); i.e., 
evaluation for subsequent decisionmaking and project 
stimulation* 
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ISSUES OF PROGRAM OR OVRIUtfX ACTiytTY EVALUATION 

1. Alternatives concerning ghat is evaluated ; 

(1) Quality of research supported. 

(2) Relevance of research supported to ongoing 
programs and interests of the entire institute. 

(3) Covering of gap areas of research. 

(4) Balance or mix among research areas supported; 
geographically, by discipline, and substantively. 

(5) Progress tovand overall objectives of the institute 
for the division. 

Alternative chosen for this management plan: 

More emphasis on (1), (3), and* (4) ; Quality, gap, and mix. 

2. Alternatives concerning yhp conducts the evaluation of division programs 
or branch activities: 

(1) Division Director 

(2) Higher Management ; e.g., NIE Director and/or 
National Advisory Council. 

(3) A permanent or ad hoc group of outside scientists 
who review the project selection decisions made by 
each program officer. 

(4) Institute staff of the Office of Policy, Planning, 
and Evaluation. 

(5) Combinations of the above. 
Alternative chosen for this management plan: 

(3) Permanent groups of outside scientists prepare the 
major program evaluations for the division. Additional 
evaluations may be conducted by higher management and the 
institute staff. 

3. Alternatives for the primary role of program or branch 
activity evaluation : 

(1) For subsequent selection of individual projects 

(2) For subsequent changes in the allocation priorities 

(3) For assessments of managerial competence and resulting 
personnel changes. 



(4) For cotamunlcatlon to the research community and 

to the general public on the progress and activities 
of the Institute In the pursuit of solutions vital 
to the future of education. 
Alternative chosen for this management plan: 
(2) The primary role of program evaluation is to 
determine desirable changes in the allocation 
priorities among research areas; other roles are of 
secondary importance. 
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IV. EVALUATION OF THE RECOMMENDED MANAGEMENT PLAN 



CHARACTERISTICS THAI PROMOTE THE TECHNICAL QUALITY 
OF THE R&D CONDUCTED 



Amount and qtiallty of scientific 
expertise consulted in the 
project selection process: 



Consideration given to risky 
projects in comparison to: 



Vulnerability of conflicts 
of interest regarding J 



The variant of the Mail Review 
paradigm described here provides 
the greatest number and greatest 
selectivity of expert opinion concern- 
ing each proposal selected for 
review of the alternatives examined. 
Proposals are reviewed by as many 
experts as needed. Additionally, 
the second round of mailed comments 
expose reviewers to the opinions 
of other experts before asking them 
to make their final evaluations. 

Reliance on program directors 
rather than panels of peer 
scientists for final selection of 
the proper mix of research projects 
helps reduce the possibility of 
limiting projects funded to those 
employing traditional methods and 
carried out by investigators of 
established reputation. Workshops 
and site visits also provide program 
directors with opportunities to 
assess research potential independent 
of conventional views . 

If a scientist's professional 
reputation is based on existing 
theory, he might not value a 
proposal that might threaten 
existing theory as highly as 
another whose professional status 
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Al)lllty of management to 
concentrate resources on 
selected problems of high 
priority: 



Attractiveness of management 
procedures to the R&D 
conmunity: 



is not tied to the existing 
theory > Since final project 
decisions in this management plan 
are made by program directors » 
vulnerability to this kind of 
conflict of interest is reduced. 

Reviewing proposals by evaluating 
each one separately without compari- 
son to others being evaluated 
tends to disperse research effort, 
because it is more difficult to 
give extra weight to the funding 
of proposals that are similar or 
complementary in nature. Within 
the management plan proposed here, 
group evaluation and the overall 
perspective of the program director 
allow a relatively high degree 
of concentration among resources 
of the Basic Research Division. 

Attractiveness of the variant of the 
mail review paradigms proposed here 
is limited by the following factors: 
finite term support ; low visibility 
of proposal reviewers , which limits 
the number of inquiries from poten- 
tial performers; and since mail 
reviews are not announced publicly, 
low. linkage with the R&D community. 
Visibility , communication , and 
linkage are provided outside the 
selection procedures, through the 
more direct activities of travel, 
workshops, conferences, and fre- 
quent contacts with mail reviewers. 
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Vulnerability to decisions 
based on personal favor: 



Decision power placed in the hands 
of a program director reduces the 
possibility of "backscratching'* 
among members of the scientific 
community, but requires a high 
degree of integrity for the direc- 
tors themselves. This integrity 
is ensured partly by the panel 
which convenes to review the decis- 
ions made, and guards against 
favoritism by the program directors. 
Appeal by an applicant in this 
variant of the mail review paradigm 
is difficult, but applicants may 
contact members of their evalua- 
tion team as they wish. 



OTHER CHARACEERISTICS THAT PROMOTE THE EFFECTIVENESS 
OF THE INSTITUTE 



Ability to Attract a 
Supportive const ituency : 



Participation of the research 
community in thi decisionmaking 
process helps build a supportive 
CO ns t i tue ncy in at least three 
ways: by building confidence that 
the research programs are well run, 
by building proprietary interests 
in the welfare of active programs, 
and by developing appointment to 
a decisionmaking council as a 
mark of distinction. In these 
regards, the variant of mail review 
recommended here contributes 
relatively little to the NIE's 
attractiveness to its research 
constituency because of the low 
profile of mail reviewers and 
the relatively larger amount of 



Sensitivity of the manageiaent 
process to the skills of n feu 
managers : 



Cost of the managenient 
process : 



decision powe r given to the 
program director. 

Effectiveness of the management 
procedures described here is 
f airly dependent on the acquisi- 
tion o£ competent program directors . 
Use of leading scientists as mail 
reviewers makes this job easier. 
Program directors must be capable of 
convincing researchers that they 
will get a fair hearing, and do so 
without excessive documentation and 
scientific assistance or handling 
of countless complaints . 

The average program management team 
within the Division of Basic 
Res.earch should consist of cue 
Program Director, one shared 
Assistant Program Director who 
covers two or three program areas, 
and one secretary. The average 
direct labor cost of this team 
should be roughly $45,000 per year. 
At any one time, 3-6 programs may 
be in operation. The cost of the 
permanent panels that periodically 
review the decis^^'ons of the Program 
Directors should be $10,000 per year 
or less for each of the 4-6 
discipline-oriented division-- 

branches . These costs are not — 

significantly below those of other 
systems requiring permanent review 
panels rather than mailed reviews. 
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Average time required to 
process an application for 
funding: 



The time to process a proposal 
is defined ehre as the period that 
elapses between the deadline for 
submission of an application and 
the time when the grantee knows 
whether or not he will be funded. 
In the variant <f mail review 
described here, which includes a 
second round of reviews, this 
average time may range fron four 
to six months . Relative to other 
management processes applied to 
basic research operations, this 
time period is not particularly 
long. 



